Statistical Proof

2024-06-30 0 1

Oh, Hyunzi. (email: wisdom302@naver.com)
Korea University, Graduate School of Economics.

Main References

Soch, Joram, et al. (2024). StatProofBook/StatProofBook.github.io: The Book of Statistical Proofs (Version 2023). Zenodo. https://doi.org/10.5281/ZENODO.4305949
Hogg et al. (2013). "Introduction to Mathematical Statistics" (8th Edition)
송성주, and 정명식. (2020). "수리통계학 - 제5판". 자유아카데미.

Random Variable

Definition (random experiments).

A random experiment is any repeatable procedure that results in one (i.e. random variable random variable) out of a well-defined set of possible outcomes.

The set of possible outcomes is called sample space, .
A set of zero or more outcomes is called a random event, .
A function that maps from events (formally, from random variables) to probabilities is called a probability function, .
Together, sample space, event space, and probability function characterize a random experiment.

Definition (random variable).

Informally, a real number is a random variable it it is the outcome of a random experiment random experiment.
Formally, a measurable function is a random variable it it is defined on a probability space and maps from a sample space to the real numbers using an event space and a probability function .
Broadly, a random variable is any random quantity such as a random event, a random scalar, a random vector or a random matrix.

Probability

Cumulative Distribution Function

Definition (distribution function).

A (cumulative) distribution function of a random variable is defined as If is a continuous random variable, then If is a discrete random variable, then

Proposition (properties of distribution function).

non-decreasing:
limit value:
continuity when increasing:
expectation:

Probability Mass Function

Definition (probability mass function).

A probability mass function (pmf) is the probability distribution of a discrete random variable:

Probability Density Function

Definition (probability density function).

A probability density function (pdf) of a continuous random variable is the function that satisfies

Theorem (pdf of invertible function of a continuous random vector).

Let be an random vector of continuous random variables with possible outcomes and let be an invertible and differentiable function on the support of . Then, the pdf of is given by where denotes the Jacobian matrix of and

Proof.First, we obtain cdf of : where is defined as and is the function with returns th element of , given .

Next, we substitute into the integral to obtain where Then, we have and therefore, which can also be written as □

Independence

Definition (statistical independence).

Generally speaking, random variables random variables are statistically independent if their joint probability can be expressed in terms of their marginal probabilities.

A set of discrete random variables with possible values is called statistically independent if where are joint probabilities of and are marginal probabilities.
A set of continuous random variables defined on the domains is called statistically independent if where are joint or marginal cumulative distribution functions. or equivalently, if the probability density functions exists, if

Moments

Moment-Generating Function

Definition (moment-generating function).

The moment-generating function (MGF) of a random variable is if is a random vector, then the moment-generating function is

Remark (finding moments from MGF).

Remark the Introductory Analysis > ^38dc05 Introductory Analysis > Theorem 18 (Taylor's Approximation) for for all : Then, we can write Therefore, and the -th moment of is the coefficient of in the Taylor series of , obtaining all moments of .

Proposition (MGF of function).

Let be a random variable with expected value function . Then, the moment-generating function of is equal to

Proof.By ^556231 Definition 9 (moment-generating function), we have Since , we simply substitute and obtain which completes the proof. □

Proposition (linear transformation of MGF).

Let be random vector with moment-generating function . Then, the moment-generating function of the linear transformation is given by

Proof.By ^556231 Definition 9 (moment-generating function), we have Then the MGF of is this completes the proof. □

Proposition (linear combination of MGF).

Let be independent random variables with moment-generating functions . Then, the moment-generating function of the linear combination is given by

Proof.Since we have which completes the proof. □

Characteristic Functions

Definition (characteristic function).

A characteristic function of any real-valued random variable is a function that completely defines its probability distribution.

The characteristic function of a random variable is
For a random vector ,
For a random matrix ,

Note that the characteristic function is similar to the cumulative distribution function of which also completely determines the behavior and properties of the probability distribution of the random variable . Thus we have

Proposition (CF of function).

Let be a random variable with expected value function . Then, the characteristic function of is equal to

Proof.From the definition using ^4c658d Theorem 17 (Law of unconscious statistician), we have which completes the proof. □

Remark (relationship with MGF).

Let be a characteristic function and be a moment generating function of a random variable . Then, we have by ^556231 Definition 9 (moment-generating function).

Expected Value

Theorem (Law of unconscious statistician).

Let be a random variable with possible outcomes and let be a function of this random variable.

If is a discrete random variable with probability mass function , then the expected value of is
If is a continuous random variable with probability density function , the expected value of is

see the proof here.

Theorem (Law of iterated expectation).

For given two random variables and , we have

Proof.If , and two random variables and are continuous, we have Thus the proof is complete. □

Theorem (multiplicity of expected value).

If two random variables and are independent, then the expected is multiplicative, i.e. However, if and are dependent, then the expected value is not necessarily multiplicative.

Proof.We only prove for when and are continuous. Assuming and are independent, we have Note that the second equation holds by ^8ee059 Definition 8 (statistical independence). □

Covariance

Definition (covariance).

The covariance of two random variables and is defined as the expected value of the product of their deviations from their individual expected values, i.e.

Theorem (covariance under independence).

Let and be independent random variables. Then, the covariance of and is zero. i.e.

Proof.Assume and are independent. Then, using ^750640 Theorem 19 (multiplicity of expected value), we have Then, we have which completes the proof. □

Multivariate Normal Distributions

Definition (multivariate Normal distribution).

An n-dimensional random vector is said to have (multivariate) Normal distribution with parameters and , and denoted by if its probability density is given by where and is positive definite matrix.

Lemma (MGF of multivariate normal distribution).

Let follows a multivariate normal distribution . Then, the moment-generating function moment-generating function of is

Proof.By ^556231 Definition 9 (moment-generating function), we have Since the multivariate normal distribution has a pdf of by ^4c658d Theorem 17 (Law of unconscious statistician), thus we have which completes the proof. □

Theorem (linear transformation of normal distribution).

Suppose that a random vector has a multivariate normal distribution where . Then, we have

Proof.The moment-generating function moment-generating function of a random vector is For a random vector defined as the moment-generating function is Using the result from ^5081bb Lemma 23 (MGF of multivariate normal distribution), as we have and therefore, the moment-generating function of is Therefore, the follows This completes the proof. □

Proof.Form MGF of : the MGF of is given by: which completes the proof. □

Useful Lemma

Cauchy-Schwarz Inequality

Theorem (Cauchy-Schwarz inequality).

For any random variables , we have where the equality holds if and only if there exists some and such that .

Proof.Let and be the population mean of and , respectively. Now define some function as Then, by the quadratic formula, the following must hold for the root to be exist: which equals to the cauchy-schwarz inequality of which completes the proof. □

Markov's Inequality

Theorem (Markov's inequality).

For random variable , define a function such that for every . Then, for , if exists.

Proof.First, define a set . Let be a random variable, and be a probability density function. Since , we have If , then . Thus we have By dividing the both sides by , we finally have which completes the proof. □

Chebyshev's Inequality

Corollary (Chebyshev's Inequality).

For a random variable , suppose and exists. Then we have

Proof.Let and let . Then, applying ^de4c5a Theorem 26 (Markov's inequality), we have Since and we have this completes the proof. □

Jensen's Inequality

Theorem (Jensen's inequality).

Let be a random variable with expectation value of , and a function be convex. Then we have

Proof.For convenience, assume that is twice differentiable. Then by a taylor expansion, there exists some between and that satisfies Since is convex, we have , and Thus we have by taking expectation on the both sides, which completes the proof. □

NickName

E-Mail

Website

Comments

Latest
Oldest
Hottest